Modern computing and display technologies have facilitated the development of systems for so-called “virtual reality” (VR) or “augmented reality” (AR) experiences, where digitally reproduced images or portions thereof are presented to a user in a manner where they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input, whereas an AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real world around the user.
VR and AR systems typically employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head, and thus move when the user's head moves. If the user's head motions are detected by a display system, image data being displayed can be updated to take into account the change in head pose (i.e., the orientation and/or location of user's head).
As an example, if a user wearing a head-worn display views a representation of a three-dimensional (3D) object as an image on the display and walks around an area where the 3D object appears, that 3D object can be regenerated from each additional viewpoint, giving the user the perception that they are walking around the 3D object. If the head-worn display is used to present a scene including multiple 3D objects within a virtual space (e.g., a rich virtual world), measurements of head pose can be used to regenerate the scene to match the user's dynamically changing head location and orientation and provide an increased sense of immersion in the virtual space.
Typically, a display imposes a color data fidelity or resolution on how color information is presented. Within the resolution, only one color value per pixel can be present, regardless of how many photonic events may correspond to an angular position from a user's point of view or perspective. In a case of opaque content, only a result of a closest photonic event (e.g., a closest opaque object to the user) is kept resulting in the total loss of data related to occluded events (e.g., opaque objects and/or semi-transparent objects behind the opaque object). In a case of semi-transparent content, results from photonic events (e.g., semi-transparent object and opaque objects and/or semi-transparent objects behind the semi-transparent object) are blended. In either case, if only one color value per pixel can be present, it becomes difficult to reconstruct and separate original photonic event data (e.g., semi-transparent objects and/or opaque objects) which makes it difficult to warp positions of those events correctly if each event should be moved to a different location during the warp. For example, if photonic events e0 and e1 have been blended to produce pixel p0, during warping a solution may be to move photonic event e0 to pixel p1 and photonic event e1 to pixel p2, but because the input to warping is the merge result (e.g., pixel p0), the merged pixel must be moved in its entirety to either pixel p1 or pixel p2 and cannot be split out back into its original photonic events e0 and e1 to perform the warping correctly and move photonic event e0 to pixel p1 and photonic event e1 to pixel p2 as desired.
For each frame, only image data corresponding to points of visible surfaces of objects are kept (i.e., rendered) and is referred to as “rendered content,” and image data corresponding to points of non-visible surfaces of the objects is discarded (i.e., not rendered) and is referred to as “non-rendered content.” However, when the user moves their head to a new perspective, the user may be expecting to see points of surfaces of the object that were previously non-visible. As explained in further detail below, any non-rendered content is lost when the user moves to a new perspective because the application has kept only rendered content from a previous scene. This means that when the VR and AR systems display different perspectives, each additional scene needs to be regenerated from each new perspective to accurately depict each scene with correct locations and orientations for the points of surfaces of the objects.
“Warping” may be applied to change the way that objects are presented in a new scene. This approach takes objects, and shifts the way that points of surfaces of the objects are presented in an attempt to display the objects from a different perspective. However, a disocclusion problem occurs when some objects are at least partially covering other objects. In one example, a frame including image data is rendered from a first perspective, where the rendered frame includes displayable information of a first object and a second object from the first perspective and does not include displayable information of the first object and the second object from a second perspective. Rendered content corresponding to displayable information for the first and second objects from the first perspective is captured and stored. Non-rendered content corresponding to displayable information for the first and second objects from the second perspective is not captured or stored when the frame is rendered from the first perspective. This becomes problematic when the user moves to a new position because different portions of the first object and the second object may be viewable from the second perspective, but warped content only includes portions of the first and second objects viewable from the first perspective. The warped content will be missing content for the first and second objects that were not visible in the first perspective but that are now visible from the second perspective (no matter how much warping is applied). This approach therefore creates a significant amount of video artifacts that affect the content veracity of the image from different perspectives.
Therefore, there is a need for an approach to display a new perspective of an image by using image data for non-rendered content from a previously rendered image without regenerating the image from the new perspective.